fix(ci): Build python wheels using interpreters in manylinux2014#2486
Merged
Conversation
* Introduce `ci/build_linux_wheels.py` to manage Linux wheel builds. * Remove the shell script `ci/build_manylinux_wheel.sh`. * Update `.github/workflows/build-wheels.yaml` to use the new Python script. * Define manylinux and musllinux images directly in the Python script. * Simplify the build process by removing explicit image environment variables. * Enhance flexibility by allowing multiple images per architecture.
esafak
changed the title
* Remove the generic `build-wheels.yaml` workflow. * Introduce `build-native-release.yml` for building native wheels on macOS and Windows for releases. * Rename `build-wheels-for-pr.yaml` to `build-native-pr.yml` and update its triggers and matrix. * Rename `build-wheels-for-release.yaml` to `build-containerized-pr.yml` and update its triggers and matrix. * Create `build-containerized-release.yml` for building containerized wheels on Linux for releases. * Update `release-python.yaml` to depend on the new native and containerized release workflows. * Modify `ci/run_ci.sh` to use `curl` for downloading Bazel and to adjust the Bazel installation directory. * Update `ci/build_linux_wheels.py` to allow specifying Python versions and to add a `--release` flag for full test suites. * Adjust Bazel job count calculation in `ci/run_ci.sh` to be based on the OS variable.
esafak
changed the title
* Remove explicit Bazel installation step from `.github/workflows/build-containerized-pr.yml`. * The `ci/run_ci.sh` script now includes a `ls "$BAZEL_DIR"` command for better visibility. * Added `git config --global --add safe.directory /work` to `ci/build_linux_wheels.py` to prevent Git errors in containerized environments.
…ci_manylinux)
* Improve Bazel verification in `run_ci.sh` by adding detailed checks:
* `ls -l` for the Bazel executable.
* `file` command to determine file type.
* `head -n 1` for initial content inspection.
* `readelf -l` to inspect ELF interpreter information.
* `strings` to find the loader path if `readelf` is unavailable.
* `ldd` to identify potential library load errors.
* Modify `deploy.sh` to prioritize Python from `$PYTHON_PATH` for `pyarrow` and `numpy` installations, falling back to default `python` if not set.
* Update `build_linux_wheels.py` to explicitly `export PYTHON_PATH` for each Python version tested, ensuring consistent usage within the script.
* Use PYTHON_PATH for python and pip commands if available. * Remove musllinux images from build_linux_wheels.py as bazel binaries are not compatible.
* Removed extensive and verbose checks for Bazel installation in `ci/run_ci.sh`. * Simplified the output message in `ci/build_linux_wheels.py` from "Testing with" to "Using" for preinstalled Python interpreters. * Added printing of the PATH environment variable in `python/setup.py` for better debugging. * Removed an unnecessary blank line in `python/setup.py`.
esafak
changed the title
esafak
changed the title
chaokunyang
reviewed
Aug 19, 2025
chaokunyang
approved these changes
Aug 19, 2025
Collaborator
chaokunyang
left a comment
chaokunyang
left a comment
There was a problem hiding this comment.
LGTM, just left a minor comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The existing workflow did not build wheels correctly because the python interpreter installed in the workflow was not used in the container where the actual building was taking place. The manylinux images already have every python version pre-installed. We now use them instead.
Key changes include:
New Workflows:
build-containerized-pr.yml: Handles building wheels for pull requests using containerized environments for Linux.build-containerized-release.yml: Handles building wheels for releases using containerized environments for Linux.build-native-pr.yml: Handles building wheels for pull requests on macOS and Windows.build-native-release.yml: Handles building wheels for releases on macOS and Windows.Removed Workflows:
.github/workflows/build-wheels.yaml: This generic workflow has been superseded by the new, more specific workflows..github/workflows/build-wheels-for-pr.yaml: Merged intobuild-containerized-pr.ymlandbuild-native-pr.yml..github/workflows/build-wheels-for-release.yaml: Merged intobuild-containerized-release.ymlandbuild-native-release.yml..github/workflows/build-native-pr.yml: This was a duplicate and is now handled by the newbuild-native-pr.yml.Script Improvements:
ci/build_linux_wheels.py: Introduced a new script to manage Linux wheel building within Docker containers, replacingbuild_manylinux_wheel.sh. This script now uses explicit Docker image definitions and a simplified execution model.ci/deploy.sh: Updated to usePYTHON_PATHandPIP_CMDfor better Python environment management, and adjustedpyarrowinstallation versions for different Python versions.ci/run_ci.sh: Updated to correctly handle Bazel installation paths (avoiding sudo) and usecurlfor downloading Bazel, since wget is not in the container.Workflow Triggers:
pushtomainor tags for releases,pull_requestevents for PR builds).Related issues
Closes #2480
Notes
I also attempted to create musl builds but they failed because the bazel binaries are incompatible with it. We would need to rebuild bazel for musllinux.